Purpose

The ultimate goal of the CKO is to ensure that the agency’s practitioners have access to critical knowledge when they need it, now and in the future, to increase the likelihood of mission success. However, in most cases, raw data is not necessarily knowledge. To acquire knowledge, the data has to be processed and analyzed in order to extract meaningful and actionable elements from it. For more information visit https://knowledge.jsc.nasa.gov.

In order to demonstrate how raw data can be transformed into knowledge, shown below is a case study based on crew debriefs that illustrates how meaningful knowledge can be extracted from textual debrief data.

Disclaimer

! WARNING - CRIMINAL PENALTIES!

Some of the information contained in this debrief may be subject to the Privacy Act of 1974, as amended. The content may be disclosed, discussed and shared with individuals when they have a direct need-to-know in the performance of their official duties. Disclosure of Agency records which contain individually identifiable information is prohibited. Individuals with access to this information by virtue of their official position who willfully disclose it to anyone not entitled to receive it may be found guilty of a misdemeanor and fined up to $5000.

Please do not distribute attributable comments without permission/review by OPSHAB & CB. Please contact Susan Schuh for further information - susan.v.schuh@nasa.gov 281-483-7487

Confirm all data and analysis with approved sources before making mission or program decisions.

Crew Perspective Case Study

Data Description

At the completion of each International Space Station (ISS) expedition, the ISS Program office schedules a series of intensive debriefs with returning U.S. crew members and International Partner crew members to determine individual observations and concerns related to working and living on board the ISS.

An estimated 20 to 25 post-mission debriefs are conducted after each expedition; these debriefs address topics related to specific disciplines and systems, including habitability and human factors, logistics and maintenance, payloads, stowage, food, procedures, etc. (Schuh et al. 2011)

Knowledge

The debriefs have resulted in approximately 80,000 crew comments, with each comment averaging roughly 114 words; the equivalent of over 90 copies of Harper Lee’s To Kill a Mocking Bird. The quantity and raw text format makes human comprehension difficult to answer questions such as:

  • How do the astronauts view various topics and systems?
  • What are the most important issues affecting space habitability?
  • How have perceptions on various topics and systems changed over time?
  • Can we automatically summarize content?
  • Can we search and extract specific content or recommendations?

Sentiment

Sentiment Analysis or opinion mining refers to the use of natural language processing, text analysis and computational linguistics to identify and extract subjective information in source materials. Generally speaking, sentiment analysis aims to determine the attitude of the speaker with respect to some topic or the overall contextual polarity of a text. The attitude may be his or her judgment or evaluation, affective state, or the intended emotional communication. (Wikipedia 2016)

The equation used by the algorithm to assign value to polarity of each comment or sentence first utilizes the sentiment dictionary (Liu 2004) to tag polarized words. A context cluster \(x_{i}^{T}\) of words is pulled from around this polarized word (default 4 words before and two words after) to be considered as valence shifters. The words in this context cluster are tagged as neutral \(x_{i}^{0}\), negator \(x_{i}^{N}\), amplifier \(x_{i}^{a}\), or de-amplifier \(x_{i}^{d}\). Neutral words hold no value in the equation but do affect word count \((n)\). Each polarized word is then weighted based on the weights from the polarity.frame argument and then further weighted by the number and position of the valence shifters directly surrounding the positive or negative word. This research used a default value of 0.8 for the weight to be utilized with amplifiers/de-amplifiers. Last, these context clusters \(x_{i}^{T}\) are summed and divided by the square root of the word count \((\sqrt{n})\) yielding an unbounded polarity score \((\delta)\). Note that context clusters containing a comma before the polarized word will only consider words found after the comma. (Rinker 2013)



\(\delta = \frac{x_{i}^{T}}{\sqrt{n}}\)

where
\(x_{T}^{i} = \sum{((1+c(x_{i}^{A}-x_{i}^{D})) * w(-1)^{\sum{x_{i}^{n}}}})\)

\(x_{i}^{A} = \sum({w_{neg} \cdot x_{i}^{a}})\)

\(x_{i}^{D} = max(x_{i}^{D'}, -1)\)

\(x_{i}^{D'} = \sum(-{w_{neg} \cdot x_{i}^{a} + x_{i}^{d}})\)

\(w_{neg} = (\sum{x_{i}^{N}}) \mod 2\)



Once we have a numeric score that corresponds to the sentiment expressed in each comment, it’s possible to prepare the data and construct any number of visualizations in order to answer a variety of questions.

Here’s an example various reports (some are subsets of data) which shows comment sentiment in various different ways.

The following plot shows the quantity of comments broken down by number of negative and positive for each primary category for a frustrating dataset. The plot is interactive and hovering over a category will allow you to compare the raw number of negative versus positive comments. It can also indicate which questions are most being asked and discussed, though care needs to be taken in this case since the comments are subsetted to those related to frustration.


Here is an example that illustrates severity for the different comments. The color indicates whether the median of the polarity of the comments is negative, neutral, or positive. Hovering over a box will provide information such as the median, inner quartiles, outer quartiles, and outliers.


The following plot represents the overall sentiment expressed by each individual comment. The size of the bubbles represent the predicted magnitude of the sentiment of the comment with larger bubbles being a stronger sentiment and smaller bubbles being more neutral. The color represents the type of sentiment with red shades expressing more negative sentiment and blue shades expressing more positive sentiment. The full text of the comment can be viewed by hovering over a particular comment; another version of this chart allows for highlighting comments and displaying them in a table below the plot.


Here is a LOESS regression showing the comment polarity over time for the ‘warnings and alarms’ category on training data. The dip in sentiment for more recent increments indicates the possibility of some negative events on station such as an increase in the number of false alarms though more investigation would be required to determine that.



Here is an example of how even text can be colored to display the sentiment expressed in text. To protect privacy this comment is not from the astronaut debriefs but still illustrates the capability.


“Equipment malfunctions will also occur, particularly during subsystem development testing. In manned flight we must regard every malfunction, and, in fact, every observed peculiarity in the behavior or a system as an important warning of potential disaster. Only when the cause is understood and a change to eliminate it has been made and verified, can we proceed with the flight program. ”
-F.J. Bailey Jr. NASA


Cluster Analysis

In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis which seeks to build a hierarchy of clusters.

Document I like hate ISS
\(D_{1}\) 1 1 0 1
\(D_{2}\) 1 0 1 1

We can then weight the documents

\(tfidf(t,d,D) = tf(t,d) \cdot idf(t,D)\)

Where:

\(tf(t,d)\) is the raw frequency of a term in a document

\(idf(t,D) = log[\frac{N}{1 + |d \in D: t \in d |}]\)

\(N\): total number of documents in the corpus \(N = \begin{vmatrix} D \end{vmatrix}\)

\(|d \in D: t \in d |\) is the number of documents where the term \(t\) appears

In statistics, Ward’s method is a criterion applied in hierarchical cluster analysis. Ward’s minimum variance method inaccurate, see talk is a special case of the objective function approach originally presented by Joe H. Ward, Ward suggested a general agglomerative hierarchical clustering procedure, where the criterion for choosing the pair of clusters to merge at each step is based on the optimal value of an objective function. Many of the standard clustering procedures are contained in this very general class. To illustrate the procedure, Ward used the example where the objective function is the error sum of squares, and this example is known as Ward’s method or more precisely Ward’s minimum variance method.

The initial cluster distances in Ward’s minimum variance method are therefore defined to be the squared Euclidean distance between points:

\({\displaystyle d_{ij}=d(\{X_{i}\},\{X_{j}\})={\|X_{i}-X_{j}\|^{2}}.}\)


Here are the actual comments from the cluster highlighted above; it should be very apparent that this cluster of negative comments is due to complaints regarding noise on station:

Lorem ipsum dolor sit amet, consect experienced high noise levels Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum perspiciatis unde omnis.Iste natus error sit voluptatem accusantium noise behind the SM panels, beatae vitae dicta sunt explicabo nemo enim ipsam.

Natus error sit voluptatem WHC fan noise, illo inventore veritatis et quasi architecto beatae vitae dicta sunt resident tone . The one on orbit doesn’t sound At vero eos et accusamus et iusto odio. Et harum quidem rerum grinding noise hic tenetur a sapiente delectus, ut aut hum or er. At vero eos et accusamus et iusto.

dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia noise constraint uod maxime placeat facere possimus, omnis uod maxime noise clock running, Nam libero tempore. Et harum quidem rerum facilis est et expedita noise in the Lab consectetur, adipisci absorbs sound; ut et silent. sed quia non numquam eius modi noise vel eum iure reprehenderit qui in ea voluptate velit esse. Nemo enim ipsam voluptatem quia voluptas sit aspernatur. maxime placeat facere possimus, omnis voluptas assumenda est, making noise.



Computer Vision

With the recent push to capture crew comments and feedback in the form of video responses, the following section is a short technology demonstration of how computer vision and supervised machine learning could supplement crew comments by automating the process of capturing emotion. This demonstration below is not very useful from an analysis perspective, but does serve to demonstrate the technology. While the particular numbers are debated (see Mehrabian & Wiener 1967 or Mehrabian & Ferris, 1967), the consensus among psychologist and cognitive researchers is that non-verbal communication makes up a non-trivial aspect of communication and attitudes therefore facial emotion is one avenue to explore when seeking to improve sentiment models.

There are numerous algorithms and approaches to identifying both faces and facial landmarks (geometric methods, eigenface methods, Gabor wavelets, discrete cossinus transforms, local binary patterns); this demonstration uses the Project Oxford algorithms (Microsoft Cognitive Services). The emotion classification is performed using supervised machine learning on facial landmark data, and a confidence score is given for each emotion.

The image below shows a surprised Chris Hadfield with markers indicating the facial landmarks detected via computer vision techniques. The table shows the predicted emotion based on the location of these facial landmarks.